Color matching across multiple sensors in an optical system

ABSTRACT

Matching color information in an optical system can include splitting an image forming beam into a bright intensity beam and a dark intensity beam, detecting, using multiple sensors, a color value for a light component from the bright intensity beam and the dark intensity beam, determining color values for the remaining light components associated with the bright intensity beam and the dark intensity beam, and transforming the color values associated with the dark intensity beam to calibrate the color values of the dark intensity beam against the color values of the light intensity beam, the color values of the light intensity beam including color inaccuracies.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Application No. 62/294,820,filed on Feb. 12, 2016, the content of which is hereby incorporated byreference in its entirety.

TECHNICAL FIELD

This disclosure relates to color matching across multiple sensors in anoptical system.

BACKGROUND

A human visual system is capable of identifying and processing visualfeatures with high dynamic range. For example, real-world images thathave contrast ratios of 1,000,000:1 or greater can be accuratelyprocessed by the human visual cortex. However, most image acquisitiondevices are only capable of reproducing or capturing low dynamic range,resulting in a loss of image accuracy. The problem is ever moresignificant in video imaging.

There are numerous examples of creating high dynamic range images bypost processing images from multiple sensors, each subject to differentexposures. The resulting “blended” image is intended to capture abroader dynamic range than would be possible from a single sensorwithout a post-processing operation.

Typical color sensors have a color filter array (CFA) incorporated withthe sensor. In a CFA sensor, each pixel has one of three color filtersplaced in front of it: Red, Green, or Blue. As is well-understood in thefield of color sensor implementation, the raw image produced by a CFAsensor thus has a characteristic Bayer pattern. In a process that iswell-understood in the field of color sensor implementation, such Bayerpattern images from color sensors are typically demosaiced to give eachpixel three unique values: a Red (R) value, a Green (G) value, and aBlue (G) value. The demosaiced image is then typically color corrected(CC) to match the values of true colors in the real world (so-called“truth” or “true” color values). The CC process is well-understood inthe field of color sensor implementation. The CC process entails firstcapturing a Bayer image of a calibrated color-square card, demosaicingthis image, and then calculating a Color Correction Matrix (CCM). TheCCM is stored in the camera's memory, and is then automaticallymultiplied by each pixel's RGB values in subsequent images, in order tocorrect subsequent images from the color corrected space (CCS) of thesensor to CCS of “truth”.

In a typical HDR merging process, multiple CC images are combinedtogether to create an HDR image.

An implicit assumption in all previous HDR merging methods is that theCCM of all sensors is identical.

Despite their presence in the dynamic range, the resulting images stillfail to reproduce the robust natural color of their subjects. Onesignificant problem with traditional attempts to improve dynamic rangeis the inability to replicate colors which are often diluted orexaggerated as a result of traditional image processing. There is aneed, therefore, for improved image processing techniques as exemplifiedby the following disclosure.

SUMMARY

In an optical system, merging information from multiple sensors improvesthe dynamic range of image capture. Methods of the invention generate atrue color image by sequentially converting data from a first sensorinto the color correction space of a second sensor, combining image datafrom the first and second sensor, and then optionally converting thecombined data into the color correction space of a third sensor andcombining the previously-combined image data with the image data fromthe third sensor, and so forth for n sensors, where n is any integernumber greater than one. Ultimately, the combined color data arecolor-corrected to generate a true color image file that is reflectiveof the dynamic range of the imaged object. The stepwise data conversionfrom multiple sensors (typically starting with the sensor receiving theleast amount of light) corrects for differences in the amount of lightreceived by sensors at different locations in the optical path used toproduce an image. The image information from multiple sensors havingbeen “equalized”, the resulting combination of information from allsensors accurately conveys light levels and color values of the originalobject being imaged.

The present invention improves on previous HDR merging methods, byconverting one sensor's data from its CCS to the CCS of another sensor(the color matching step), before merging the images from the twosensors. The merged image data can then be converted to the CCS of athird sensor, before being combined with the image data from that thirdsensor. The process may be repeated for as many sensors as desired.After all sensors' images have been combined, the final combined imagemay be demosaiced and then may be color corrected to truth.

The purpose of the color matching step in the present invention is tocorrect for any differences in light spectrum between light thatreflects and light that transmits through the optical system's variousbeamsplitters. Although such beamsplitters may be called “spectrallyneutral”, there are often slight wavelength-dependent differences in theamount of light transmitted. These differences can cause the amounts(and/or ratios) of R, G, and B colors sensed by the various sensors, fora single color of input light, to be different.

Methods described herein are also useful to correct for the variation inthe detected color levels among different sensors that is caused by theposition of the sensor relative to a beam splitter (e.g., glass cubes orpellicles). Beamsplitters typically include a partially reflectivesurface (e.g., a thin film coating). As a light beam either reflectsfrom or transmits through a beamsplitter, properties of the light beam(e.g., transmittance or reflection) are affected. As a light beamtravels through multiple beamsplitters, the effects increaseexponentially. In a multiple sensor optical system, combining colorinformation without accounting for these effects will prevent the systemfrom capturing a true color HDR image. By transforming all of thedetected color information into the corrected color space of a singlesensor, color data are readily combinable to produce a true color image.In addition, the described methods are also not affected by variationsin the reflective coating due to, for example, lot to lot variations orchemical reformulations.

In addition, methods described herein facilitate color matching thevarious sensors' arrays to each other without excessive processing timebecause, at least in part, the color matching process is based on pixelinformation rather than frame information. Because color matchingprocesses of the invention do not require information for a completeframe, they can be integrated into a real time HDR pipeline. In otherwords, color matching processes of the invention are compatible with anHDR optical system that processes pixel level information independent ofoverall frame information. In addition, the methods described herein arecompatible with hardware already used in HDR image or video systems, sono additional hardware is required.

These and other aspects, features, and implementations, and combinationsof them may be expressed as apparatus, methods, methods of doingbusiness, means or steps for performing functions, components, systems,program products, and in other ways.

Other aspects, features, and advantages will be apparent from thedescription and drawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Features and advantages of the claimed subject matter will be apparentfrom the following detailed description of embodiments consistenttherewith, which description should be considered with reference to theaccompanying drawings.

FIG. 1 is a schematic illustrating an exemplary HDR optical system.

FIG. 2 is a schematic of an exemplary arrangement of three sensors andtwo pellicles for use with the system shown in FIG. 1.

FIG. 3 is a schematic of an exemplary arrangement of two sensors and acube for use with the system shown in FIG. 1.

FIG. 4 is a graph illustrating the transmittance of light through a beamsplitting object.

FIG. 5 is a graph illustrating the variation in color detection betweentwo sensors within a system.

FIG. 6 is a flow chart showing the flow of information through thesystem shown in FIG. 1.

FIG. 7 is a flow chart of an exemplary color matching process.

FIG. 8 is a block diagram of a computing system for use with the systemshown in FIG. 1.

For a thorough understanding of the present disclosure, reference shouldbe made to the following detailed description, including the appendedclaims, in connection with the above-described drawings. Although thepresent disclosure is described in connection with exemplaryembodiments, the disclosure is not intended to be limited to thespecific forms set forth herein. It is understood that various omissionsand substitutions of equivalents are contemplated as circumstances maysuggest or render expedient.

DETAILED DESCRIPTION

The invention relates to correcting deviations in detected color in anoptical system with two or more sensors where the deviations are, atleast in part, caused by sensor to sensor variation. The sensors (e.g.,CCD sensors, CMOS sensors, focal plane arrays, or other light detectionsensors) include an array of light detection elements (e.g., pixels).The invention teaches matching color detection values across multiplesensors and eventually color correcting the color values withoutprematurely (e.g., before color correction) losing saturated dataindicators from the multiple sensors.

An exemplary process utilizing the invention to create an HDR image fromsensor data from two CFA sensors by converting the data from a firstsensor from its CCS to another sensor's CCS includes:

-   -   1) a first sensor (called the SE sensor) that receives a small        portion of the incoming light and a second sensor (called the ME        sensor) that receives a larger portion of the incoming light.        For example, with a 90:10 beamsplitter between the two sensors,        the ME sensor might receive approximately nine times as much        light as the SE sensor.    -   2) a color card, to calculate the CCM for converting the SE        sensor's image to truth. This CCM is called [C] below;    -   3) a color card to measure the CCM for converting the ME        sensor's image to truth. This CCM is called [B] below. Note that        because the ME sensor receives more light than the SE sensor,        the ME sensor will probably require less light on the color card        than was used with the SE sensor, in order to be well-lit. That        is why there are often two stages to the CCM calculation;    -   4) an algorithm to demosaic the SE image data;    -   5) a processor to multiply SE image pixel data by the product of        the two matrices, [C] and [B]⁻¹ shown below; where the notation        “[B]⁻¹” represents the inverse of matrix [B];    -   6) the demosaic process is then reversed (that is, mosaic the        image into a Bayer-patterned image). This yields a        non-demosaiced SE image, whose colors have been converted to the        ME sensor's CCS, which image is called the “SE (in ME CCS)”        image; and    -   7) a processor to merge the newly-formed, Bayer-patterned “SE        (in ME CCS)” image with the ME image (which is still in its raw,        Bayer-patterned state).        This merged, Bayer-patterned image may now be demosaiced, and        then color corrected by simply multiplying its pixels by the [B]        matrix.

Although the example above shows how to combine two sensors' data, steps4 through 7 may be repeated as many times as necessary to accommodatecombining images from any number of sensors.

Another exemplary process utilizing the invention to create an HDR imagefrom sensor data from two CFA sensors by converting the data from afirst sensor from its CCS to another sensors CCS includes utilizing acolor correction lookup table (LUT). In particular, step 5 above may bereplaced with the following steps:

-   -   5a) a processor to transform the SE image pixel data into true        colors in the real world (so-called “truth” or “true” color        values) by performing a lookup of an SE sensor's color        correction LUT.    -   5b) using the true colors, the processor further performing a        reverse lookup of an ME sensor's color correction LUT to produce        a demosaiced SE image, whose colors have been converted to the        ME sensor's CCS. Alternatively, using the true colors, the        processor performs a lookup in a color matching LUT (i.e., an        inverted ME sensor color correction LUT) to produce the        demosaiced image.

The SE image is then merged according to steps 6-7 above. This merged,Bayer-patterned image may now be demosaiced, and then color corrected byusing the ME sensor's color correction LUT.

Advantageously, the LUT process reduces processing resources as requiredby the CCM approach.

FIG. 1 shows an HDR optical system 100, including an HDR sub-system 104.When light 108 enters a main lens 106 of the HDR optical system 100, thelight 108 is detected by the sensor and beam splitter components 107.The HDR optical system 100 can also include a processor 110 and software114 for collecting and analyzing data detected by the sensor and beamsplitter components 107 and a memory 112 for storing (long-term ortemporary) information.

Referring to FIGS. 1 and 2, the sensors and beam splitter components 107can be arranged in pellicle configuration 200 including three sensors202 a-c. In this example, each of the sensors 202 a-c is exposed tolight for substantially the same period. The sensor 202 c is sometimesreferred to as a large exposure (LE) sensor because it is positionedwhere it will receive the greatest amount of illumination. The sensor202 b is sometimes referred to as a small exposure (SE) sensor becauseit is positioned where it will receive the least light amongst thesensors. The sensor 202 a is sometimes referred to as the mediumexposure (ME) sensor because it will receive light between the brightestlight and the darkest light received amongst the sensors.

In this example, the light 108 travels in directions generally indicatedby an arrow 204. For example, the light travels unhindered for a firstdistance 212 before reaching a first pellicle 208. Upon reaching thefirst pellicle 208, a portion of the light transmits through the firstpellicle 208 and a portion is reflected away from the first pellicle208. The transmitted light continues over a path 214 before reaching theLE sensor 202C. The reflected light continues over a path 216 beforereaching a second pellicle 210. Upon reaching the second pellicle 210, aportion of the light transmits through the second pellicle 210 and aportion is reflected away from the second pellicle 210. The transmittedlight travels along a path 218 before reaching the ME sensor 202 a. Thereflected light travels along a path 220 before reaching the SE sensor202 b.

Referring to FIGS. 1 and 3, the sensors and beam splitter components 107can be arranged in a cube configuration 300. Like the pellicleconfiguration 200, in the cube configuration 300, each of the sensors202 b-c experience substantially the same amount of light exposure time.In this example, the light 108 travels along a first optical path 306through part of a prism 302 before reaching a partially-reflectivecoating 304. Upon reaching the partially-reflective coating 304, aportion of the light transmits through the partially-reflective coating304, and a portion of the light 108 is reflected away from by thepartially-reflective coating 304. The transmitted light continues alongan optical path 308 before reaching the LE sensor 202 c, and thereflected light continues along an optical path 310 before reaching theSE sensor 202 b. Other sensor and beam splitter configurations can alsobe used such as those disclosed in U.S. Pat. Nos. 8,320,047, 8,441,732,and 8,619,368, the contents which are each incorporated by reference intheir entirety.

As shown in FIGS. 4 and 5, a beam splitting object (e.g., the pellicle,cube, or a reflective coating), will affect the detected properties oflight. For example, FIG. 4 shows a graph 400 illustrating how thetransmittance percent of light within the beam splitting object behaves.The light does not travel unperturbed through the object, theunperturbed path being indicated by a dashed line 402. Instead thetransmittance percent, for example, will oscillate as a function oflight wavelength (or color) within the coating as shown by line 404.This oscillation can produce discrepancies between color values detectedby different sensors based on where the information is detected in theoscillation. In addition, as shown in FIG. 5, a first sensor may detectcolor values which indicate a transmittance percentage represented by aline 502. However, due to the non-direct path within the beamsplittersor due to losses caused by multiple beamsplitters on the optical path, asecond sensor may detect color values that indicate a transmittancepercentage represented by a line 504. For images or videos displayed atlower dynamic ranges, the differences 506 between the line 502 and theline 504 may be inconsequential. However, in HDR image and videocapture, these discrepancies can produce an image with noticeable colorerrors. Color matching between the various sensors can help to addressthis issue.

Referring to FIG. 6, the flow of information in an HDR optical system100 is shown. In this example, information (e.g., values from one ormore pixels indicative of a detected color intensity) from each of thesensors 202 a-c is transmitted to a sync module 602 which synchronizesthe pixel signals. The pixel signals are then sent to a color matchmodule 604 that executes a color matching process 700 (as shown in FIG.7). After completing the color matching process 700, the color valuesare transmitted to a color correction module 606 that executes a colorcorrection process 707. The color corrected information can thenoptionally undergo additional processing by, for example, a tone mappingoperator (TMO) module 605 before being transmitted to a display 608.

Referring to FIG. 7, the color matching process 700 for the HDR opticalsystem 100 having the pellicle configuration 200 is shown. The purposeof the color matching process 700 is to merge images from multiplesensors together, in stages where two images are merged at a time, in away that preserves color information from one sensor to the next. As anexample, in FIG. 7, the darkest image is shown, from the SE sensor,being merged with the middle-brightness image, from the ME sensor. Theresult of this merger is then merged with the brightest image, from theLE sensor.

The basic principle guiding the color matching process 700 is to firstconvert a dark image to the color correction space of the next brightestimage, and then to merge the two “non-demosaiced” (or Color Filter Array[CFA] Bayer-patterned) images together.

The color matching process 700 for the pellicle configuration 200, whichhas 3 sensor arrays 202 a-c, comprises three general phases: an SE colorcorrection space (CCS) phase 701, ME CCS phase 703, and LE CCS phase705. The color matching process first begins with the SE CCS phase 701,which comprises first demosaicing 702 the CFA Bayer-patterned imageproduced by the SE sensor 202 b, and then matching (or converting) 704the CCS of this SE image to the CCS of the ME image. The demosaicingprocess 702 yields a full-color value to each pixel in the image.

After it has been demosaiced 702, the SE image data is next matched intothe ME CCS 703. The purpose of this conversion step 704 is to match thecolor of the SE array image to the color of the ME array (with all ofthe ME array's color imperfections).

In one example, to perform this conversion 704, the pixel input signalsare transformed through a color correction matrix. For example,Equations 1-3 show the color correction matrices for the LE, ME, and SEsensors, respectively. The color correction matrix for the LE sensor iscalculated using Equation 1, where the 3×3 matrix coefficients,including values A₁-A₉, represent coefficients selected to strengthen orweaken the pixel value, and an RGB matrix (R_(LE), G_(LE), and B_(LE))represents the demosaiced RGB output signal from the LE sensor. In somecases, the 3×3 matrix coefficients can be derived by comparing thedemosaiced output against expected (or so-called “truth”) values. Forexample, the 3×3 matrix coefficients can be derived by least-squarespolynomial modeling between the demosaiced RGB output values andreference values from a reference color chart (e.g., a Macbeth Chart).Similarly, the color correction matrix for the ME sensor is calculatedusing Equation 2, where the RGB matrix (R_(ME), G_(ME), and B_(ME))represents the demosaiced RGB output signal from the ME sensor, and thecolor correction matrix for the SE sensor is calculated using Equation3, where the RGB matrix (R_(ME), G_(ME), and B_(ME)) represents thedemosaiced RGB output values from the SE sensor.

$\begin{matrix}{{{correcting}\mspace{14mu}{SE}\mspace{14mu}{pixel}{\mspace{11mu}\;}{values}\mspace{14mu}{{using}\mspace{14mu}\lbrack A\rbrack}},\;{{the}{\mspace{11mu}\;}{Color}\mspace{14mu}{Correction}\mspace{14mu}{Matrix}\mspace{14mu}{for}\mspace{14mu}{the}\mspace{14mu}{LE}\mspace{14mu}{sensor}}} & \; \\{{\begin{bmatrix}A_{1} & A_{2} & A_{3} \\A_{4} & A_{5} & A_{6} \\A_{7} & A_{8} & A_{9}\end{bmatrix}\begin{bmatrix}R_{LE} \\G_{LE} \\B_{LE}\end{bmatrix}} = {{\lbrack A\rbrack\begin{bmatrix}R_{LE} \\G_{LE} \\B_{LE}\end{bmatrix}} = \begin{bmatrix}R_{truth} \\G_{truth} \\B_{truth}\end{bmatrix}}} & {{Equation}\mspace{14mu} 1} \\{{{correcting}\mspace{14mu}{ME}\mspace{14mu}{pixel}\mspace{14mu}{values}\mspace{14mu}{{using}\mspace{14mu}\lbrack B\rbrack}},{{the}{\mspace{11mu}\;}{Color}\mspace{14mu}{Correction}\mspace{14mu}{Matrix}\mspace{14mu}{for}\mspace{14mu}{the}\mspace{14mu}{ME}\mspace{14mu}{sensor}}} & \; \\{{\begin{bmatrix}B_{1} & B_{2} & B_{3} \\B_{4} & B_{5} & B_{6} \\B_{7} & B_{8} & B_{9}\end{bmatrix}\begin{bmatrix}R_{ME} \\G_{ME} \\B_{ME}\end{bmatrix}} = {{\lbrack B\rbrack\begin{bmatrix}R_{ME} \\G_{ME} \\B_{ME}\end{bmatrix}} = \begin{bmatrix}R_{truth} \\G_{truth} \\B_{truth}\end{bmatrix}}} & {{Equation}\mspace{14mu} 2} \\{{{correcting}\mspace{11mu}{SE}\mspace{14mu}{pixel}\mspace{11mu}{values}\mspace{11mu}{{using}\mspace{14mu}\lbrack C\rbrack}},{{the}\mspace{11mu}{Color}\mspace{11mu}{Correction}\mspace{11mu}{Matrix}\mspace{11mu}{for}\mspace{11mu}{the}\mspace{11mu}{SE}\mspace{11mu}{sensor}}} & \; \\{{\begin{bmatrix}C_{1} & C_{2} & C_{3} \\C_{4} & C_{5} & C_{6} \\C_{7} & C_{8} & C_{9}\end{bmatrix}\begin{bmatrix}R_{SE} \\G_{SE} \\B_{SE}\end{bmatrix}} = {{\lbrack C\rbrack\begin{bmatrix}R_{SE} \\G_{SE} \\B_{SE}\end{bmatrix}} = \begin{bmatrix}R_{truth} \\G_{truth} \\B_{truth}\end{bmatrix}}} & {{Equation}\mspace{14mu} 3}\end{matrix}$

To convert an image from a first color correction space (CCS) to asecond CCS, the color correction matrices from one or more sensors canbe used. This process may be referred to as converting between CCSs orcalibrating CCSs. Neither the first CCS nor the 10 second CCS accuratelyreflects the true color of the captured image—both the first and thesecond CCS both have color inaccuracies, and these inaccuracies are, ingeneral, different from one another. The fact that each sensor requiresa unique CM multiplication in order to correct its color to “truth”bears out this fact. The present invention provides a method for takingthe image from a first sensor and (1) removing its color inaccuracies,and then (2) introducing the color inaccuracies from the second sensor.

The two-step process 704 for converting an image from the SE sensor'sCCS to the ME sensor's CCS is illustrated in Equation 4 below:

$\begin{matrix}{{converting}\mspace{14mu}{SE}\mspace{14mu}{pixel}\mspace{14mu}{values}\mspace{14mu}{from}\mspace{14mu}{SE}\mspace{14mu}{CCS}\mspace{14mu}{to}\mspace{14mu}{ME}\mspace{14mu}{CCS}} & \; \\{\mspace{155mu}{{{\begin{bmatrix}R_{SE} \\G_{SE} \\B_{SE}\end{bmatrix}\lbrack C\rbrack}\lbrack B\rbrack}^{- 1} = \begin{bmatrix}R_{ME} \\G_{ME} \\B_{ME}\end{bmatrix}}} & {{Equation}\mspace{14mu} 4}\end{matrix}$

In Equation 4, the SE sensor's pixel values (R, G, B) are multiplied bythe SE sensor's Correction Matrix, [C], and then multiplied by theinverse of the ME sensor's Correction Matrix, [B]. The result is a setof pixel values (R, G, B) that are in the ME sensor's CCS.

In another example, to perform this conversion 704, the pixel inputsignals are transformed using a color correction lookup table (LUT). Forexample, the color correction matrices of Equations 1-3 can be used tocreate lookup tables for the LE, ME, and SE sensors, respectively. Forinstance, the color correction LUT for the LE sensor can be generated bymultiplying all possible input RGB LE signals by the CCM of Equation 1.Accordingly, the LUT for the LE sensor maps all possible input RGB LEvalues to corresponding true color values. Similarly, the colorcorrection LUT for the ME sensor is generated by multiplying allpossible input RGB ME signals by the CCM of Equation 2. Additionally,the color correction LUT for the SE sensor is generated by multiplyingall possible input RGB SE signals by the CCM of Equation 3. Each of theLUTs can be stored in memory (e.g., the memory 820 of FIG. 8) or a massstorage device (e.g., the storage device 830 of FIG. 8). In someembodiments, each of the LUTs can be combined into a single colorcorrection LUT.

The process 704 for converting an image from the SE sensor's CCS to theME sensor's CCS is as follows. The SE sensor's pixel values (R, G, B)are used to perform a lookup using the SE sensor's lookup table. The SEsensor's lookup table outputs corresponding true (R, G, B) color values.The true (R, G, B) color values of the SE sensor are then used to obtaincolors in the ME sensor's CCS. This is accomplished by using the true(R, G, B) color values to perform a lookup using the ME sensor's lookuptable. In one aspect, the ME sensor's lookup table can be a reverse LUTgenerated from the color correction LUT for the ME sensor by, forexample, performing an inversion of the ME sensor's color correctionLUT. Alternatively, a reverse lookup can be performed on the ME sensor'scolor correction LUT. This process yields a set of pixel values (R, G,B) that are in the ME sensor's CCS.

Methods of the invention allow matching of the CCS of the second sensorto the CCS of the first sensor so that the images from the two sensorsmay be accurately combined, or merged. The method for applying all theinaccuracies of the second CCS to the first CCS, prior to combiningimages from the two into an HDR image, is previously unknown. Typicalmethods for combining data from multiple CFA sensors rely oncolor-correcting each sensor's data to the “truth” values measured froma calibrated color card, prior to combining the images. This isproblematic in an HDR system, where it is known that the brightersensor's image will have significant portions that are saturated, whichsaturated portions should actually have been taken from the darkersensor's image when combining. Color correcting an image that hassaturated pixels will cause colors to be miss-identified. Therefore, inan HDR system, combining images that have been individuallycolor-corrected to “truth” values will almost always lead to colorsbeing miss-identified.

The solution provided in the present invention avoids this colormisidentification problem by performing the steps of [(a) demosaic, (b)color match & (c) mosaic] only on data that comes from the darkersensor, thereby ensuring all data is accurately returned to itsnon-demosaiced state prior to the step of merging the darker sensor'sdata with the brighter sensor's data.

Furthermore, prior to merging the images from two sensors, the presentinvention matches the Color Correction Spaces of the two sensors. Thistransformation is ensures that the two images (from the first and secondCCS sensors) can be accurately merged, pixel-for-pixel, innon-demosaiced format. It may at first seem counterintuitive to changethe CCS of one sensor to match the CCS of a second sensor, especiallywhen the second sensor's CCS is known to differ from the “true” CCS.However, it is an important step in ensuring the color data from bothsensors is matched together, prior to merging the images.

After the SE information is transformed from the SE CCS 701 to the MECCS 703, the transformed values are mosaiced 706 (i.e., the demosaicingprocess is reversed). The transformed SECFA Bayer-patterned pixel valuesare now comparable with the CFA Bayer-patterned pixel values detected bythe ME sensor, which supports a seamless merge 708 of ME and SEnon-demosaiced sensor data.

The merged non-demosaiced image within the ME CCS 703 can now undergo ademosaicing process 710. This demosaicing process is similar to thedemosaicing process 702 described above, except the CFA pixel valuesundergoing the demosaicing process are now associated with the ME CCS.

Still Referring to FIG. 7, to transform 712 the color values associatedwith the ME CCS, an ME color correction matrix can be determined usingthe demosaiced color values, as shown above in Equation 2. The colorcorrected ME information is transformed from the ME CCS 703 to the LECCS 705 by multiplying the ME color correction matrix by the inverse ofthe LE color correction matrix, as shown in Equation 4.

Alternatively, to transform 712 the color values associated with the MECCS, the color values are converted into true colors from the ME CCS byperforming a lookup in the ME sensor's color correction LUT. The true(R, G, B) color values produced by the ME sensor's LUT are then used toobtain colors in the LE sensor's CCS. This is accomplished by using thetrue (R, G, B) color values produced by the ME sensor's LUT to perform alookup using the LE sensor's color correction lookup table. In oneaspect, the LE sensor's lookup table can be a reverse LUT generated fromthe LE sensor's color correction LUT. Alternatively, a reverse lookupcan be performed on the LE sensor's color correction LUT. This processyields a set of pixel values (R, G, B) that are in the LE sensor's CCS.

After the ME information is transformed from the ME CCS 703 to the LECCS 705, the transformed values are mosaiced 714 (i.e., the demosaicingprocess is reversed). This allows the transformed ME CFA Bayer-patternedpixel values to merge 716 with the CFA Bayer-patterned pixel valuesdetected by the LE sensor. At this point in the color matching process700, the transformed color information detected by the SE and ME sensorsis now calibrated to match the color information detected by the LEsensor. This newly merged color value data set now represents colorvalues within the LE color correction space 705.

Still referring to FIG. 7, after the color matching process 700 iscomplete, the color values detected by the various sensors are nowcomparable with each other. However, as discussed above, these colorvalues still include color inaccuracies, specifically these color valuesare in the LE CCS, and therefore require correction using the LE sensorCM, called [A]. Alternatively, correction can be performed by using theLE sensor's LUT. These color inaccuracies would produce an image whichdoes not match the color of the captured object. To color correct forthese inaccuracies (also referred to as correcting to truth), a colorcorrecting process 707 can be used. In the color correcting process 707,the color values associated with the LE color correction space (bothtransformed and detected) undergo a demosaicing process 718, which isgenerally similar to the demosaicing processes 702, 710 of the colormatching process 700. To correct for any color value inaccuracies, theLE color correction matrix (as defined by Equation 1) can be multipliedby the LE pixel values ([LE]) to correct the LE color values toaccurately reflect the color values of the original captured image, asshown in Equation 4. Alternatively, correction can be performed by usingthe LE sensor's LUT. This yields an image with color information thatnot only excludes inaccuracies introduced by the multiple sensors butalso matches the color information of “truth”. A final image can then begenerated, and/or displayed, and the final image includes color levelsrepresentative of the captured object.

The ME CCS process 703, which comprises the steps of conversion of SEinformation to ME CCS 704, mosaic 706, merge 708, and demosaic 710, maybe optionally removed or performed additionally, in order to accommodateany number of multiple sensors. For example, for a system comprising twosensors in a cube configuration 300, the ME CCS process 703 may beomitted from the color matching process 700, so that the color matchingprocess 700 comprises the SE CS 701 and LE CCS 705 steps, correspondingto the SE sensor array 202 b and LE sensor array 202 c of the cubeconfiguration 300.

FIG. 8 is a block diagram of an exemplary computer system 800. Forexample, referring to FIG. 1, the HDR sub-system 104 can include thesystem 800. The system 800 includes a processor 810, a memory 820, astorage device 830, and an input/output device 940. Each of thecomponents 810, 920, 930, and 940 can be interconnected, for example,using a system bus 850. The processor 810 is capable of processinginstructions for execution within the system 800. In someimplementations, the processor 810 is a single-threaded processor. Insome implementations, the processor 810 is a multi-threaded processor.In some implementations, the processor 810 is a quantum computer. Insome implementations, the processor 810 is a Field Programmable GateArray (FPGA). In some implementations, the processor 810 is anApplication Specific Integrated Circuit (ASIC). The processor 810 iscapable of processing instructions stored in the memory 820 or on thestorage device 830. The processor 810 may execute operations such as thecolor matching or color correcting operations described herein.

The memory 820 stores information within the system 800. In someimplementations, the memory 820 is a computer-readable medium. In someimplementations, the memory 820 is a volatile memory unit. In someimplementations, the memory 820 is a non-volatile memory unit.

The storage device 830 is capable of providing mass storage for thesystem 800. In some implementations, the storage device 830 is acomputer-readable medium. In some implementations, the storage device830 can include, for example, a hard disk device, an optical diskdevice, a solid-state drive, a flash drive, magnetic tape, or some otherlarge capacity storage device. In some implementations, the storagedevice 830 may be a cloud storage device, e.g., a logical storage deviceincluding multiple physical storage devices distributed on a network andaccessed using a network. The input/output device 940 providesinput/output operations for the system 800. In some implementations, theinput/output device 940 can include one or more of a network interfacedevices, e.g., an Ethernet card, a serial communication device, e.g., anRS-232 port, and/or a wireless interface device, e.g., an 802.9 card, a3G wireless modem, a 4G wireless modem, or a carrier pigeon interface.In some implementations, the input/output device can include driverdevices configured to receive input data and send output data to otherinput/output devices, e.g., keyboard, printer and display devices 860.In some implementations, mobile computing devices, mobile communicationdevices, and other devices can be used.

While certain implementations have been described above, otherimplementations are possible. For example, while the color matchingprocess has generally been described as progressing from an SE CCS to anME CCS to an LE CCS, other sequences are possible. For example, theprogression can be reversed, the progression may proceed directly fromthe SE CCS to the LE CCS, or may proceed in any order through two ormore correction spaces.

While the color matching process has generally been described as usingthree sensors (e.g., as shown in FIG. 7), the color matching process mayalso be used with only two sensors or with more than three sensors. Inthese cases, the general methodology of the three sensors embodiment canbe expanded or reduced to accommodate the desired sensor configurationso long as the pixels from desired sensors remain at least partiallyaligned physically.

The term “system” may encompass all apparatus, devices, and machines forprocessing data, including by way of example a programmable processor, acomputer, or multiple processors or computers. A processing system caninclude, in addition to hardware, code that creates an executionenvironment for the computer program in question, e.g., code thatconstitutes processor firmware, a protocol stack, a database managementsystem, an operating system, or a combination of one or more of them.

A computer program (also known as a program, software, firmware,software application, script, executable logic, or code) can be writtenin any form of programming language, including compiled or interpretedlanguages, or declarative or procedural languages, and it can bedeployed in any form, including as a standalone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment or for implementation in a Field Programmable Gate Array(FPGA) or compiled firmware embedded in an Application SpecificIntegrated Circuit (ASIC). A computer program does not necessarilycorrespond to a file in a file system. A program can be stored in aportion of a file that holds other programs or data (e.g., one or morescripts stored in a markup language document), in a single filededicated to the program in question, or in multiple coordinated files(e.g., files that store one or more modules, sub-programs, or portionsof code). A computer program can be deployed to be executed on onecomputer or on multiple computers that are located at one site ordistributed across multiple sites and interconnected by a communicationnetwork.

Computer readable media suitable for storing computer programinstructions and data include all forms of non-volatile or volatilememory, media and memory devices, including by way of examplesemiconductor memory devices, e.g., EPROM, EEPROM, and flash memorydevices; magnetic disks, e.g., internal hard disks or removable disks ormagnetic tapes; magneto-optical disks; CD-ROM and DVD-ROM disks; or anyother type of non-transitory storage medium suitable for storingelectronic instructions, and each coupled to a computer system bus. Theprocessor and the memory can be supplemented by, or incorporated in,special purpose logic circuitry. The components of the system can beinterconnected by any form or medium of digital data communication,e.g., a communication network. Examples of communication networksinclude a LAN and a WAN.

INCORPORATION BY REFERENCE

References and citations to other documents, such as patents, patentapplications, patent publications, journals, books, papers, webcontents, have been made throughout this disclosure. All such documentsare hereby incorporated herein by reference in their entirety for allpurposes.

EQUIVALENTS

Various modifications of the invention and many further embodimentsthereof, in addition to those shown and described herein, will becomeapparent to those skilled in the art from the full contents of thisdocument, including references to the scientific and patent literaturecited herein. The subject matter herein contains important information,exemplification, and guidance that can be adapted to the practice ofthis invention in its various embodiments and equivalents thereof.

What is claimed is:
 1. A method for matching color information in anoptical system, the method comprising: splitting an image forming beaminto two or more beams comprising at least a bright intensity beam and adark intensity beam; detecting, using multiple sensors, a color valuefor a light component from the bright intensity beam and the darkintensity beam; determining color values for the remaining lightcomponents associated with the bright intensity beam and the darkintensity beam; and transforming the color values associated with thedark intensity beam to calibrate the color values of the dark intensitybeam against the color values of the bright intensity beam, the colorvalues of the bright intensity beam including color inaccuracies.
 2. Themethod of claim 1, further comprising color correcting the transformedcolor values and the color values associated with the bright intensitybeam for color inaccuracies using reference values.
 3. The method ofclaim 1, wherein the bright intensity beam is detected by a first sensorand the dark intensity beam is detected by a second sensor.
 4. Themethod of claim of claim 1: wherein splitting the image forming beamfurther comprises splitting the image forming beam into a third mediumintensity beam; wherein detecting the color value for the lightcomponent further comprises detecting the light component from themedium intensity beam; wherein determining the color values furthercomprises determining color values for the remaining light componentsassociated with the medium intensity beam; wherein transforming thecolor values to calibrate the color values of the dark intensity beamagainst the color values of the bright intensity beam further comprises:transforming the color values associated with the dark intensity beam tocalibrate the color values of the dark intensity beam against the colorvalues of the medium intensity beam, the color values of the mediumintensity beam including color inaccuracies; and transforming the colorvalues associated with the medium intensity beam and the calibrated darkintensity beam color values to calibrate the color values against thecolor values of the bright intensity beam, the bright intensity beamincluding color inaccuracies.
 5. The method of claim 4, furthercomprising color correcting the color values associated with the brightintensity beam and the color values calibrated against the brightintensity beam for color inaccuracies using reference values.
 6. Themethod of claim 4, wherein the bright intensity beam is detected by afirst sensor, the medium intensity beam is detected by a second sensor,and the dark intensity beam being detected by a third sensor.
 7. Amethod for matching color information in an optical system, the methodcomprising: detecting light information from each beam split of an imageforming beam, the image forming beam being split to impinge each sensorof a plurality of sensors, each of the sensors having a plurality ofpixels and each of the sensors receiving a different amount of lightsuch that the detected light information of each beam split from each ofthe sensors is associated with a different color correction space;sequentially transforming the detected light information at each sensorfrom a first color correction space to a target color correction spaceuntil the detected light information from each of the sensors isassociated with a final color correction space; merging the detectedlight information and transformed light information associated with thefinal color correction space from each sensor; color correcting themerged light information to correct for color inaccuracies usingreference values.
 8. The method of claim 7, wherein each of theplurality of pixels is covered by a color filter such that the detectedlight information at each sensor includes a detected intensity value fora red, blue, or green light component associated with each pixel.
 9. Themethod of claim 7, wherein sequentially transforming the detected lightinformation at each sensor includes transforming the detected lightinformation at each sensor from a darkest color correction space to alightest color correction space until the detected light informationfrom each of the sensors is associated with the lightest colorcorrection space.
 10. The method of claim 9, wherein the final colorcorrection space is the lightest color correction space.
 11. The methodof claim 7, wherein the plurality of sensors includes three sensors. 12.A method for performing color matching on image signals, the methodcomprising: splitting an image forming beam into a two or more beams ofdiffering intensity; detecting, using a plurality of sensors, a colorvalue for a light component from each of the two or more beams;performing color correction on pixel data produced by a pixel of a firstsensor of the plurality of sensors using the color value and a colorcorrection lookup table; converting the color corrected pixel data to acolor space of a second sensor of the plurality of sensors using a colormatching lookup table; and merging the converted pixel data with pixeldata produced by a pixel of the second sensor.
 13. The method of claim12, wherein the converted pixel data is in a demosaic state and furthercomprising transforming the converted pixel data into a mosaic state.14. The method of claim 13 wherein merging includes merging theconverted pixel data in its mosaic state with the pixel data produced bythe pixel of the second sensor, wherein the pixel data produced by thepixel of the second sensor is also in a mosaic state.
 15. The method ofclaim 14 further comprising: transforming the merged pixel data into ademosaic state; and performing color correction on the merged pixel datausing the color correction lookup table.
 16. The method of claim 15further comprising: converting the color corrected merged pixel data toa color space of a third sensor of the plurality of sensors using thecolor matching lookup table; merging the converted merged pixel datawith pixel data produced by a pixel of the third sensor into a finalmerged pixel output; and performing color correction on the final mergedpixel output using the color correction lookup table.
 17. The method ofclaim 12 wherein the color correction lookup table includes referencesvalues used to remove color inaccuracies from pixel data produced byeach pixel of each sensor of the plurality of sensors.
 18. The method ofclaim 12 wherein the color matching look up table includes values forconverting color corrected pixel data into a color space of any sensorof the plurality of sensors.